Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution][Threat Intelligence] - fix Cypress tests #209195

Merged
merged 2 commits into from
Feb 3, 2025
Merged

[Security Solution][Threat Intelligence] - fix Cypress tests #209195

merged 2 commits into from
Feb 3, 2025
+30 −33

Conversation

PhilippeOberti
Copy link
Contributor

@PhilippeOberti PhilippeOberti commented Feb 1, 2025 •
edited by kibanamachine
Loading

Summary

This PR fixes the few Threat Intelligence Cypress tests which started failing recently.

This PR (commit on main) which was a basic bump PR started had to skip a few Threat Intelligence Cypress tests that started failing.

I tried to understand what happened but did not find the origin of the issue. The indicators table was previously showing data sorted from oldest to newest (but only on Cypress, not when running the application locally, really weird). After the commit mentioned above, the Cypress tests are now sorting the data from newest to oldest.
Looking at the code, nothing mentions sorting, the threatIntelligenceSearchStrategy is identical between the application running locally and the Cypress tests.

Payload

Local run Cypress
Screenshot 2025-01-31 at 2 20 33 PM Screenshot 2025-01-31 at 2 21 39 PM

Result

The results are different, one sorted ascending, the other descending.

Local run Cypress
Screenshot 2025-01-31 at 2 22 19 PM Screenshot 2025-01-31 at 2 22 03 PM

No code changes have been done to the Threat Intelligence plugin in many months. Fetching the data is done using useQuery (see here) and the logic to fetch the actual data is here. Both haven't been touched in months...

Nothing in the commit mentioned above seems to give any details on why these tests just started failing. Resetting to the commit right before makes the tests pass...

The solution to fix the tests was to ensure that the data_archive files had the correct timestamps, to know exactly which ones we select in the Cypress tests to test against...

#209050
#209039
#209051

@PhilippeOberti PhilippeOberti added release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting:Investigations Security Solution Investigations Team backport:version Backport to applied version labels v9.1.0 v8.19.0 labels Feb 1, 2025
@PhilippeOberti PhilippeOberti requested review from a team as code owners February 1, 2025 00:16
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)

@elasticmachine
Copy link
Contributor

elasticmachine commented Feb 1, 2025 •
edited
Loading

💔 Build Failed

Failed CI Steps

Metrics [docs]

✅ unchanged

History

MadameSheema
MadameSheema approved these changes Feb 3, 2025
View reviewed changes
Copy link
Member

@MadameSheema MadameSheema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-engineering-productivity LGTM!

Qq @PhilippeOberti is there any reason why these tests are executed in ESS and not in serverless? Thanks! :)

@janmonschke
Copy link
Contributor

janmonschke commented Feb 3, 2025 •
edited
Loading

@PhilippeOberti Given that BASELINE_ELASTICSEARCH_VERSION was also bumped to 9.1.0 makes me think that the issue came from a change in ES itself. I wonder if we should dig deeper?

Edit: I had a quick look at the changes that were merged to ES 9.1.0 and nothing out of the ordinary popped up.

@janmonschke
Copy link
Contributor

@elasticmachine merge upstream

@PhilippeOberti PhilippeOberti merged commit 4886061 into elastic:main Feb 3, 2025
9 checks passed
@PhilippeOberti PhilippeOberti deleted the fix-TI-cypress-tests branch February 3, 2025 15:03
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13116417385

@PhilippeOberti
Copy link
Contributor Author

PhilippeOberti commented Feb 3, 2025 •
edited
Loading

security-engineering-productivity LGTM!

Qq @PhilippeOberti is there any reason why these tests are executed in ESS and not in serverless? Thanks! :)

@MadameSheema I just have not had the time to test them in serverless...

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 3, 2025
@PhilippeOberti
[Security Solution][Threat Intelligence] - fix Cypress tests (elastic…
4c30608 
…#209195)

## Summary

This PR fixes the few Threat Intelligence Cypress tests which started
failing recently.

[This PR](elastic#208990)
([commit](elastic@296c452)
on `main`) which was a basic bump PR started had to skip a few Threat
Intelligence Cypress tests that started failing.

I tried to understand what happened but did not find the origin of the
issue. The indicators table was previously showing data sorted from
oldest to newest (but only on Cypress, not when running the application
locally, really weird). After the commit mentioned above, the Cypress
tests are now sorting the data from newest to oldest.
Looking at the code, nothing mentions sorting, the
`threatIntelligenceSearchStrategy` is identical between the application
running locally and the Cypress tests.

#### Payload

| Local run  | Cypress |
| ------------- | ------------- |
| ![Screenshot 2025-01-31 at 2 20
33 PM](https://github.com/user-attachments/assets/7095eeef-3ceb-4a3c-85ee-7fee7e07b9ba)
| ![Screenshot 2025-01-31 at 2 21
39 PM](https://github.com/user-attachments/assets/7bce82d2-12ff-483f-bf85-ad8f0ce45054)
|

#### Result

The results are different, one sorted ascending, the other descending.

| Local run  | Cypress |
| ------------- | ------------- |
| ![Screenshot 2025-01-31 at 2 22
19 PM](https://github.com/user-attachments/assets/09715344-0c6e-44e9-8abd-d0b56ae8c984)
| ![Screenshot 2025-01-31 at 2 22
03 PM](https://github.com/user-attachments/assets/b66412fc-0018-4a84-9ddf-98b90c200878)
|

No code changes have been done to the Threat Intelligence plugin in many
months. Fetching the data is done using `useQuery` (see
[here](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/threat_intelligence/public/modules/indicators/hooks/use_indicators.ts#L101))
and the logic to fetch the actual data is
[here](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/threat_intelligence/public/modules/indicators/services/fetch_indicators.ts#L71).
Both haven't been touched in months...

Nothing in the commit mentioned above seems to give any details on why
these tests just started failing. Resetting to the commit right before
makes the tests pass...

The solution to fix the tests was to ensure that the data_archive files
had the correct timestamps, to know exactly which ones we select in the
Cypress tests to test against...

elastic#209050
elastic#209039
elastic#209051

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 4886061)
@kibanamachine kibanamachine mentioned this pull request Feb 3, 2025
Closed
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@PhilippeOberti
Copy link
Contributor Author

@PhilippeOberti Given that BASELINE_ELASTICSEARCH_VERSION was also bumped to 9.1.0 makes me think that the issue came from a change in ES itself. I wonder if we should dig deeper?

Edit: I had a quick look at the changes that were merged to ES 9.1.0 and nothing out of the ordinary popped up.

@janmonschke I thought about that too, but decided to not look into it more as it seems that nobody else is complaining about this... Though this seems like a breaking change to me, if it really is that ES is returning the results ordered in a different way...

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 4, 2025
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

4 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

1 similar comment
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 209195 locally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janmonschke janmonschke janmonschke approved these changes

@MadameSheema MadameSheema MadameSheema approved these changes

Assignees
No one assigned
Labels
backport missing Added to PRs automatically when the are determined to be missing a backport. backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting:Investigations Security Solution Investigations Team v9.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@PhilippeOberti @elasticmachine @janmonschke @kibanamachine @MadameSheema